Virtual Object Structures and Interrelationships

ABSTRACT

A virtual object system can orchestrate virtual objects defined as a collection of components and with inheritance in an object hierarchy. Virtual object components can include a container, data, a template, and a controller. A container can define the volume the virtual object is authorized to write into. A virtual object’s data can specify features such as visual elements, parameters, links to external data, meta-data, etc. The template can define view states of the virtual object and contextual breakpoints for transitioning between them. Each view state can control when and how the virtual object presents data elements. The controller can define logic for the virtual object to respond to input, context, etc. The definition of each object can specify which other object in an object hierarchy that object extends, where extending an object includes inheriting that object’s components, which can be modified or overwritten as part of the extension.

TECHNICAL FIELD

The present disclosure is directed to the creation and orchestration ofvirtual objects, on in artificial reality system, defined as acollection of components and with inheritance in an object hierarchybetween virtual objects.

BACKGROUND

Interactions with computing systems are often founded on a set of coreconcepts that define how users can interact with that computing system.For example, early operating systems provided textual interfaces tointeract with a file directory. This was later built upon with theaddition of “windowing” systems, whereby levels in the file directoryand executing applications were displayed in multiple windows, eachallocated a portion of a 2D display that was populated with contentselected for that window (e.g., all the files from the same level in thedirectory, a graphical user interface generated by an application, menusor controls for the operating system, etc.). As computing form factorsdecreased in size and added integrated hardware capabilities (e.g.,cameras, GPS, wireless antennas, etc.) the core concepts again evolved,moving to an “app” focus where each app encapsulated a capability of thecomputing system. New artificial reality systems have providedopportunities for further object and interaction models.

Existing artificial reality systems provide virtual objects, such as 3Dvirtual objects and 2D panels, with which a user can interact in 3Dspace. Existing artificial reality systems have generally backed thesevirtual objects by extending the app core computing concept. Forexample, a user can instantiate these models by activating an app andtelling the app to create the model, and using the model as an interfaceback to the app. Such existing artificial reality systems are highlyunintuitive, inflexible, and difficult to create content for. Forexample, existing artificial reality systems typically limit virtualobjects to be used by the app that created them, require each user tolearn how to use the virtual objects crated by each app, and makevirtual object development labor intensive and prone to error.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an overview of devices on whichsome implementations of the present technology can operate.

FIG. 2A is a wire diagram illustrating a virtual reality headset whichcan be used in some implementations of the present technology.

FIG. 2B is a wire diagram illustrating a mixed reality headset which canbe used in some implementations of the present technology.

FIG. 2C is a wire diagram illustrating controllers which, in someimplementations, a user can hold in one or both hands to interact withan artificial reality environment.

FIG. 3 is a block diagram illustrating an overview of an environment inwhich some implementations of the present technology can operate.

FIG. 4 is a block diagram illustrating components which, in someimplementations, can be used in a system employing the disclosedtechnology.

FIGS. 5A and 5B are conceptual diagrams illustrating a breakdown of anobject into components.

FIG. 6 is a conceptual diagram illustrating an object in 3D spacedisplayed according to a view state from a template.

FIG. 7 is a conceptual diagram illustrating a container component of anobject and an example implementation of the container component.

FIG. 8 is a conceptual diagram illustrating a data component of anobject and an example implementation of the data component.

FIGS. 9A and 9B are conceptual diagrams and illustrating a templatecomponent of a person object with multiple view states and an exampleset of contextual breakpoint transitions between the view states.

FIG. 10 is a conceptual diagram illustrating a controller component ofan object and an example implementation of the controller component.

FIG. 11 is a conceptual diagram illustrating an example objecthierarchy.

FIG. 12 is a conceptual diagram illustrating an example native objectsportion of an object hierarchy.

FIG. 13 is a conceptual diagram illustrating an example description of abase object.

FIG. 14 is a conceptual diagram illustrating an example description of anative person object.

FIG. 15 is a conceptual diagram illustrating an example description of anative place object.

FIG. 16 is a conceptual diagram illustrating an example description of anative photo object.

FIG. 17 is a conceptual diagram illustrating an example description of anative conversation object.

FIG. 18 is a conceptual diagram illustrating an example description of anative post object.

FIG. 19 is a flow diagram illustrating a process used in someimplementations for defining a virtual object with a modular designprocess.

FIG. 20 is a flow diagram illustrating a process used in someimplementations for constructing a reusable template.

The techniques introduced here may be better understood by referring tothe following Detailed Description in conjunction with the accompanyingdrawings, in which like reference numerals indicate identical orfunctionally similar elements.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed to a virtual objectsystem for creating an orchestrating virtual objects in an artificialreality environment. The virtual object system defines how content canbe written into a volume in an artificial reality environment, rules forshowing that content with various contexts, and connections and linksbetween virtual objects. For example, virtual objects created andorchestrated in the virtual object system can have a common structure ofcomponents including a container, data, a template, and a controller.Such a virtual object system benefits both virtual object creators andend-users by systematizing the design patterns of virtual objects, sothat they’ll be consistent across virtual object sources. This can makevirtual object use more intuitive with expected behaviors, virtualobject design faster and easier with reusable content, and virtualobject functionality more flexible and dynamic with the ability toinherit features from other objects in a hierarchy.

A virtual object’s container can define the volume the virtual object isauthorized to write into. In some cases, the container can be a set ofnested containers, e.g., an outer container for the entire virtualobject which can include A) a data container to hold the displayelements of the virtual object (selected and organized according to acurrent view state as discussed below) and B) zero or more chromecontainers to hold UI elements associated with the virtual object. Insome implementations, a virtual object’s container is provided by acontrolling application (e.g., operating system, shell, or otherapplication in control of the artificial reality environment) to anapplication requesting to create a virtual object. The application cansubmit a manifest to the controlling application defining details of thecontainer needed and can receive back a handle to the container havingbeen assigned permissions to write, by way of the handle, into thevolume that the container occupies. Additional details on containers(also referred to as augments) are provided in U.S. Pat. Application No.17/008,478, filed on Aug. 31, 2020, titled “ARTIFICIAL REALITY AUGMENTSAND SURFACES,” which is hereby incorporated by reference in itsentirety.

A virtual object’s data can specify features of the virtual object suchas visual elements, parameters, links to external data, meta-data, etc.In some cases, the virtual object’s data can be stored as a link to alive data source, allowing the virtual object to present dynamic data(e.g., updated as the data in the live data source is updated). Forexample, a person virtual object may show an avatar of the person andhave a mood data item, specified in a cloud service of a social mediaplatform, controlling an expression worn by the avatar. As the mood dataitem is set according to user input, such as a post to the social mediaplatform defining the user’s mood, the virtual object can be updatedwith the current value for the mood data item and accordingly set theavatar’s expression. While there is an infinite variety of possible dataand data types a virtual object can have, types of virtual object canspecify required data that must be supplied to a constructor whencreating the virtual object. For example, a person virtual object mayrequire a name, an avatar, a status, etc. In some cases, various of thevirtual object data items can be inherited from a parent object (e.g.,an object from which the virtual object is defined as an extension, asdiscussed below in relation to a virtual object hierarchy). In somecases, virtual object data can be retrieved from a specified sourcebased on a controlling identifier. For example, a social media postvirtual object can be defined to have a number of visual elements,parameters, etc., but the post virtual object can be defined bysupplying a social media platform and an ID for the post. Uponinstantiation of the post virtual object, an artificial reality devicecan retrieve the virtual object’s data elements by querying thespecified social media platform for the needed data elements that areassociated with the defined post ID.

The template component can define multiple view states of the virtualobject and contextual breakpoint rules for transitioning between them.Each view state can control what and how the virtual object presentsdata (e.g., visual elements, auditory output, effects) and what thevirtual object can use as an anchor point while in that state (e.g.,horizontal surfaces, vertical surfaces, points floating in space, etc.)In some cases, the template can be defined as a state machine with eachview state being a state of the state machine and the contextualbreakpoints defining, for each state, the conditions for transitioningto another state. In various implementations, a template can be definedfor a particular virtual object, can be a reusable template defined fora category of virtual object and selected for the particular virtualobject, or can be inherited from a parent virtual object in thehierarchy (as discussed below).

The controller component can define logic for the virtual object torespond to input (e.g., user actions, world state / context, and outputsfrom system services / applications, etc.) For example, the controllercan define virtual object rules for object behaviors, spatial behaviors,component management, data management, social actions, notificationactions, privacy actions, system actions, etc. In some cases, thecontroller can define which services the virtual object hooks up to andhow the virtual object responds to events from those services. Forexample, the controller of a person virtual object may define how thevirtual object connects to a messaging service and how the personvirtual object can display an associated conversation virtual object(which, for example, may be a compound element of the person virtualobject or a separate virtual object) upon receiving a new message inthat conversation from the messaging service. The controller can alsoinclude definitions for how objects plug into each other (e.g., acontroller of a first person virtual object can define a connection to asecond person virtual object based on a social graph indicating the userassociated with the first person virtual object and the user associatedwith the second person virtual object are defined as “friends.”) Thecontroller can specify both internal logic for the virtual object andhow the virtual object will operate in different environments andthrough the controller, the virtual object can be hooked into multiplenetwork enabled services - e.g., social media platforms, news sources,weather sources, internet of things / smart home data, connections tolocal devices, messaging / calling services, mapping services, shoppingservices, transportation services, financial services, gaming systems,media providers, calendar services, storage providers, securityservices, etc. The rules specified by the controller may depend on thecurrent view state of the virtual object. The rules specified by thecontroller may also depend on any number of other factors such as whichother people or objects are in the vicinity, how the virtual object isgrouped with or linked to other objects, a state associated with thecurrent user, etc. Some of the virtual object behaviors provided by thecontroller can be inherited from a parent virtual object in thehierarchy (as discussed below). For example, a base virtual object (or“root” object) in the hierarch can define some behaviors such as beingable to peel an object out of a group or how the object reacts whendropped onto another object of a particular type. All the virtualobjects in the hierarchy can descend, though one or more levels ofinheritance, from the base virtual object and another virtual objectthat descends from the base virtual object can inherit some behaviors,except for any of these behaviors that are excluded or overwritten inthe definition of the virtual object. Some virtual object behaviors thatcan be defined and inherited are described in U.S. Pat. Application No.17/131,563, filed on Dec. 22, 2020, titled “AUGMENT ORCHESTRATION IN ANARTIFICIAL REALITY ENVIRONMENT,” which is hereby incorporated byreference in its entirety. By defining each object to have a controllerwith its own rules, each object is self-contained to only use its ownresources (i.e., the resources in scope to the virtual object).

Each virtual object instantiated in an artificial reality environmentcan be part of an object hierarchy. The definition of each object canspecify which other object in the hierarchy that object extends, whereextending an object includes inheriting that object’s components(container definition or “manifest,” data, template, and controller),which can be modified or overwritten as part of the extension. Forexample, the hierarchy can include a base object as a root node thatprovides low-level services to all extending objects. In some cases, thebase object can specify fields such as a name, ID, an owner / author ofthe object, a set of attachable Uls, a notification interface, a defaulttemplate with contextual breakpoints (e.g., for vertical, horizontal,and floating view states), interfaces with a social media social graph,position configurations (e.g., world locked, body locked, device locked,etc.), data management resources, social media actions (e.g., share,send, post, etc.), object management actions (e.g., open / close, add toobject grouping, save to collection, etc.), privacy actions, amongothers. In some implementations, a set of native objects (e.g., person,place, photo, post, event, page, and/or conversation) each extends thebase object and can be further extended by applications to createapplication versions of these native objects (“app objects”). In somecases, a special purpose object (e.g., with a container and Uls forparticular effects) extend the base object, with further special objects(e.g., clock, confetti, weather app, timer, etc.) further extending thisspecial purpose object and inheriting its container and UI elements. Yetfurther in some cases, a custom object can extend the base object, withany custom modification to the base object and further custom objectscan extend this custom object, inheriting its components.

To define a virtual object, the artificial reality system can receive adesignation of which object in the hierarchy the new virtual object willextend -specifying an initial set of components for the new virtualobject. The virtual object system can then receive designations ofadditional or alternate components to use for this virtual object. Thiscan include defining the object’s parameters (i.e., a manifest) forrequesting a container; the object’s data (e.g., visual elements, state,meta-data, attachable Uls, other objects used inside this object, etc.);features of the object’s controller (e.g., rules defining how the objectinteracts with external elements - such as other objects / services /network data, etc. - and user input), and aspects of a template for theobject (e.g., a set of view states with contextual breakpoints). In eachcase, the virtual object definition process can add to the inheritedcomponents, remove aspects of the inherited components, or replace theinherited components with alternate ones. In some cases, the templateselected for the virtual object can be from a set of reusable templates,which can be defined by specifying one or more view states, definingcontextual breakpoints among the view states, and saving the template asa reusable prefab that can be selected for the definition of multiplevirtual objects.

Embodiments of the disclosed technology may include or be implemented inconjunction with an artificial reality system. Artificial reality orextra reality (XR) is a form of reality that has been adjusted in somemanner before presentation to a user, which may include, e.g., virtualreality (VR), augmented reality (AR), mixed reality (MR), hybridreality, or some combination and/or derivatives thereof. Artificialreality content may include completely generated content or generatedcontent combined with captured content (e.g., real-world photographs).The artificial reality content may include video, audio, hapticfeedback, or some combination thereof, any of which may be presented ina single channel or in multiple channels (such as stereo video thatproduces a three-dimensional effect to the viewer). Additionally, insome embodiments, artificial reality may be associated withapplications, products, accessories, services, or some combinationthereof, that are, e.g., used to create content in an artificial realityand/or used in (e.g., perform activities in) an artificial reality. Theartificial reality system that provides the artificial reality contentmay be implemented on various platforms, including a head-mounteddisplay (HMD) connected to a host computer system, a standalone HMD, amobile device or computing system, a “cave” environment or otherprojection system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

“Virtual reality” or “VR,” as used herein, refers to an immersiveexperience where a user’s visual input is controlled by a computingsystem. “Augmented reality” or “AR” refers to systems where a user viewsimages of the real world after they have passed through a computingsystem. For example, a tablet with a camera on the back can captureimages of the real world and then display the images on the screen onthe opposite side of the tablet from the camera. The tablet can processand adjust or “augment” the images as they pass through the system, suchas by adding virtual objects. “Mixed reality” or “MR” refers to systemswhere light entering a user’s eye is partially generated by a computingsystem and partially composes light reflected off objects in the realworld. For example, a MR headset could be shaped as a pair of glasseswith a pass-through display, which allows light from the real world topass through a waveguide that simultaneously emits light from aprojector in the MR headset, allowing the MR headset to present virtualobjects intermixed with the real objects the user can see. “Artificialreality,” “extra reality,” or “XR,” as used herein, refers to any of VR,AR, MR, or any combination or hybrid thereof.

In existing artificial reality systems, virtual object are typicallysegregated within their originating application. While, in some cases,the ownership of the virtual object can be moved between applications,this typically requires advanced coordination between the applicationsso they each can know what features the virtual object has and how tointeract with it. Thus, the process of moving virtual objects betweenapplications generates friction and can break functionality or beimpossible between some applications. Further, each application being incharge of how its virtual objects react to input, what user interfacesit has, and how it hooks into system services makes existing artificialreality systems unintuitive, requiring users learn the interactionpatterns provided by each application. Yet further, existing artificialreality systems generally require application developers to build theirvirtual objects from scratch. While they may provide some modules oreditors, they fail to allow a virtual object created by one applicationto incorporate or extend virtual objects created by another application.

The virtual object system and methods described herein are expected toovercome these failing in existing artificial reality systems, providingan architecture that is more intuitive, consistent, scalable,data-centric, and social. The virtual object system makes virtualobjects more intuitive as most objects will share some base behaviorsinherited from the base object in an object hierarchy, while objectsrepresenting real-world entities can inherit features of those entitiesfrom pre-defined versions. This enables the virtual objects to easilymimic behaviors of the people, places, and things from the real world,as well as taking on behaviors to effectively function in an artificialreality environment. Virtual objects that inherit or extend data,capabilities, behaviors and views from each other in the hierarchyenhances reusability, reduces data redundancy, standardizes objectdesign patterns, and provides a foundation for data security. Further,the virtual object system makes virtual objects more consistent througheach object inheriting capabilities, data, behaviors, and views from itsparent in the object hierarchy. Users become familiar with theseelements provided by the base object and through the elements providedby type-specific native virtual objects, resulting in a consistent userexperiences with developers only having to pick an appropriate place inthe hierarchy for these inheritances. By providing an object hierarchythe virtual object system standardizes the way objects are built andrepresented, making it easier for users and developers to build aconsistent mental model of construction, interaction, and representationwith virtual objects. In addition, the consistent virtual object designand inheritances provided by the virtual object system allows virtualobjects to not be tied to a particular application, so the virtualobjects can move between contexts and objects from different sources canbe used together or within each other. The virtual object system canalso provide a scalable development platform, with objects havingconsistent structures allowing them to be combined to create morecomplex objects, or be broken down with their components formingreusable prefabs for other virtual objects. In addition, the virtualobject system can provide virtual objects that are data-centric withobjects, through their inheritances, automatically linked into datasources such as a social graph, data storage provider, media providers,etc., and automatically supporting data-centric capabilities such associal media activities, tracked user actions, event attendance,shopping services, banking, etc. For example, various virtual objectscan each represent an element on a social graph (as described below),automatically connecting the virtual object to other nodes through theedges in the social graph.

Several implementations are discussed below in more detail in referenceto the figures. FIG. 1 is a block diagram illustrating an overview ofdevices on which some implementations of the disclosed technology canoperate. The devices can comprise hardware components of a computingsystem 100 that facilitates the creation and orchestration of virtualobjects, on an artificial reality device, defined as a collection ofcomponents and with inheritance between virtual objects in an objecthierarchy. In various implementations, computing system 100 can includea single computing device 103 or multiple computing devices (e.g.,computing device 101, computing device 102, and computing device 103)that communicate over wired or wireless channels to distributeprocessing and share input data. In some implementations, computingsystem 100 can include a stand-alone headset capable of providing acomputer created or augmented experience for a user without the need forexternal processing or sensors. In other implementations, computingsystem 100 can include multiple computing devices such as a headset anda core processing component (such as a console, mobile device, or serversystem) where some processing operations are performed on the headsetand others are offloaded to the core processing component. Exampleheadsets are described below in relation to FIGS. 2A and 2B. In someimplementations, position and environment data can be gathered only bysensors incorporated in the headset device, while in otherimplementations one or more of the non-headset computing devices caninclude sensor components that can track environment or position data.

Computing system 100 can include one or more processor(s) 110 (e.g.,central processing units (CPUs), graphical processing units (GPUs),holographic processing units (HPUs), etc.) Processors 110 can be asingle processing unit or multiple processing units in a device ordistributed across multiple devices (e.g., distributed across two ormore of computing devices 101-103).

Computing system 100 can include one or more input devices 120 thatprovide input to the processors 110, notifying them of actions. Theactions can be mediated by a hardware controller that interprets thesignals received from the input device and communicates the informationto the processors 110 using a communication protocol. Each input device120 can include, for example, a mouse, a keyboard, a touchscreen, atouchpad, a wearable input device (e.g., a haptics glove, a bracelet, aring, an earring, a necklace, a watch, etc.), a camera (or otherlight-based input device, e.g., an infrared sensor), a microphone, orother user input devices.

Processors 110 can be coupled to other hardware devices, for example,with the use of an internal or external bus, such as a PCI bus, SCSIbus, or wireless connection. The processors 110 can communicate with ahardware controller for devices, such as for a display 130. Display 130can be used to display text and graphics. In some implementations,display 130 includes the input device as part of the display, such aswhen the input device is a touchscreen or is equipped with an eyedirection monitoring system. In some implementations, the display isseparate from the input device. Examples of display devices are: an LCDdisplay screen, an LED display screen, a projected, holographic, oraugmented reality display (such as a heads-up display device or ahead-mounted device), and so on. Other I/O devices 140 can also becoupled to the processor, such as a network chip or card, video chip orcard, audio chip or card, USB, firewire or other external device,camera, printer, speakers, CD-ROM drive, DVD drive, disk drive, etc.

In some implementations, input from the I/O devices 140, such ascameras, depth sensors, IMU sensor, GPS units, LiDAR or othertime-of-flights sensors, etc. can be used by the computing system 100 toidentify and map the physical environment of the user while tracking theuser’s location within that environment. This simultaneous localizationand mapping (SLAM) system can generate maps (e.g., topologies, girds,etc.) for an area (which may be a room, building, outdoor space, etc.)and/or obtain maps previously generated by computing system 100 oranother computing system that had mapped the area. The SLAM system cantrack the user within the area based on factors such as GPS data,matching identified objects and structures to mapped objects andstructures, monitoring acceleration and other position changes, etc.

Computing system 100 can include a communication device capable ofcommunicating wirelessly or wire-based with other local computingdevices or a network node. The communication device can communicate withanother device or a server through a network using, for example, TCP/IPprotocols. Computing system 100 can utilize the communication device todistribute operations across multiple network devices.

The processors 110 can have access to a memory 150, which can becontained on one of the computing devices of computing system 100 or canbe distributed across of the multiple computing devices of computingsystem 100 or other external devices. A memory includes one or morehardware devices for volatile or non-volatile storage, and can includeboth read-only and writable memory. For example, a memory can includeone or more of random access memory (RAM), various caches, CPUregisters, read-only memory (ROM), and writable non-volatile memory,such as flash memory, hard drives, floppy disks, CDs, DVDs, magneticstorage devices, tape drives, and so forth. A memory is not apropagating signal divorced from underlying hardware; a memory is thusnon-transitory. Memory 150 can include program memory 160 that storesprograms and software, such as an operating system 162, virtual objectsystem 164, and other application programs 166. Memory 150 can alsoinclude data memory 170 that can include virtual object components(e.g., containers, data, templates, controllers), virtual objecthierarchies, configuration data, settings, user options or preferences,etc., which can be provided to the program memory 160 or any element ofthe computing system 100.

Some implementations can be operational with numerous other computingsystem environments or configurations. Examples of computing systems,environments, and/or configurations that may be suitable for use withthe technology include, but are not limited to, XR headsets, personalcomputers, server computers, handheld or laptop devices, cellulartelephones, wearable electronics, gaming consoles, tablet devices,multiprocessor systems, microprocessor-based systems, set-top boxes,programmable consumer electronics, network PCs, minicomputers, mainframecomputers, distributed computing environments that include any of theabove systems or devices, or the like.

FIG. 2A is a wire diagram of a virtual reality head-mounted display(HMD) 200, in accordance with some embodiments. The HMD 200 includes afront rigid body 205 and a band 210. The front rigid body 205 includesone or more electronic display elements of an electronic display 245, aninertial motion unit (IMU) 215, one or more position sensors 220,locators 225, and one or more compute units 230. The position sensors220, the IMU 215, and compute units 230 may be internal to the HMD 200and may not be visible to the user. In various implementations, the IMU215, position sensors 220, and locators 225 can track movement andlocation of the HMD 200 in the real world and in an artificial realityenvironment in three degrees of freedom (3DoF) or six degrees of freedom(6DoF). For example, the locators 225 can emit infrared light beamswhich create light points on real objects around the HMD 200. As anotherexample, the IMU 215 can include e.g., one or more accelerometers,gyroscopes, magnetometers, other non-camera-based position, force, ororientation sensors, or combinations thereof. One or more cameras (notshown) integrated with the HMD 200 can detect the light points. Computeunits 230 in the HMD 200 can use the detected light points toextrapolate position and movement of the HMD 200 as well as to identifythe shape and position of the real objects surrounding the HMD 200.

The electronic display 245 can be integrated with the front rigid body205 and can provide image light to a user as dictated by the computeunits 230. In various embodiments, the electronic display 245 can be asingle electronic display or multiple electronic displays (e.g., adisplay for each user eye). Examples of the electronic display 245include: a liquid crystal display (LCD), an organic light-emitting diode(OLED) display, an active-matrix organic light-emitting diode display(AMOLED), a display including one or more quantum dot light-emittingdiode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER,etc.), some other display, or some combination thereof.

In some implementations, the HMD 200 can be coupled to a core processingcomponent such as a personal computer (PC) (not shown) and/or one ormore external sensors (not shown). The external sensors can monitor theHMD 200 (e.g., via light emitted from the HMD 200) which the PC can use,in combination with output from the IMU 215 and position sensors 220, todetermine the location and movement of the HMD 200.

FIG. 2B is a wire diagram of a mixed reality HMD system 250 whichincludes a mixed reality HMD 252 and a core processing component 254.The mixed reality HMD 252 and the core processing component 254 cancommunicate via a wireless connection (e.g., a 60 GHz link) as indicatedby link 256. In other implementations, the mixed reality system 250includes a headset only, without an external compute device or includesother wired or wireless connections between the mixed reality HMD 252and the core processing component 254. The mixed reality HMD 252includes a pass-through display 258 and a frame 260. The frame 260 canhouse various electronic components (not shown) such as light projectors(e.g., LASERs, LEDs, etc.), cameras, eye-tracking sensors, MEMScomponents, networking components, etc.

The projectors can be coupled to the pass-through display 258, e.g., viaoptical elements, to display media to a user. The optical elements caninclude one or more waveguide assemblies, reflectors, lenses, mirrors,collimators, gratings, etc., for directing light from the projectors toa user’s eye. Image data can be transmitted from the core processingcomponent 254 via link 256 to HMD 252. Controllers in the HMD 252 canconvert the image data into light pulses from the projectors, which canbe transmitted via the optical elements as output light to the user’seye. The output light can mix with light that passes through the display258, allowing the output light to present virtual objects that appear asif they exist in the real world.

Similarly to the HMD 200, the HMD system 250 can also include motion andposition tracking units, cameras, light sources, etc., which allow theHMD system 250 to, e.g., track itself in 3DoF or 6DoF, track portions ofthe user (e.g., hands, feet, head, or other body parts), map virtualobjects to appear as stationary as the HMD 252 moves, and have virtualobjects react to gestures and other real-world objects.

FIG. 2C illustrates controllers 270, which, in some implementations, auser can hold in one or both hands to interact with an artificialreality environment presented by the HMD 200 and/or HMD 250. Thecontrollers 270 can be in communication with the HMDs, either directlyor via an external device (e.g., core processing component 254). Thecontrollers can have their own IMU units, position sensors, and/or canemit further light points. The HMD 200 or 250, external sensors, orsensors in the controllers can track these controller light points todetermine the controller positions and/or orientations (e.g., to trackthe controllers in 3DoF or 6DoF). The compute units 230 in the HMD 200or the core processing component 254 can use this tracking, incombination with IMU and position output, to monitor hand positions andmotions of the user. The controllers can also include various buttons(e.g., buttons 272A-F) and/or joysticks (e.g., joysticks 274A-B), whicha user can actuate to provide input and interact with objects.

In various implementations, the HMD 200 or 250 can also includeadditional subsystems, such as an eye tracking unit, an audio system,various network components, etc., to monitor indications of userinteractions and intentions. For example, in some implementations,instead of or in addition to controllers, one or more cameras includedin the HMD 200 or 250, or from external cameras, can monitor thepositions and poses of the user’s hands to determine gestures and otherhand and body motions. As another example, one or more light sources canilluminate either or both of the user’s eyes and the HMD 200 or 250 canuse eye-facing cameras to capture a reflection of this light todetermine eye position (e.g., based on set of reflections around theuser’s cornea), modeling the user’s eye and determining a gazedirection.

FIG. 3 is a block diagram illustrating an overview of an environment 300in which some implementations of the disclosed technology can operate.Environment 300 can include one or more client computing devices 305A-D,examples of which can include computing system 100. In someimplementations, some of the client computing devices (e.g., clientcomputing device 305B) can be the HMD 200 or the HMD system 250. Clientcomputing devices 305 can operate in a networked environment usinglogical connections through network 330 to one or more remote computers,such as a server computing device.

In some implementations, server 310 can be an edge server which receivesclient requests and coordinates fulfillment of those requests throughother servers, such as servers 320A-C. Server computing devices 310 and320 can comprise computing systems, such as computing system 100. Thougheach server computing device 310 and 320 is displayed logically as asingle server, server computing devices can each be a distributedcomputing environment encompassing multiple computing devices located atthe same or at geographically disparate physical locations.

Client computing devices 305 and server computing devices 310 and 320can each act as a server or client to other server/client device(s).Server 310 can connect to a database 315. Servers 320A-C can eachconnect to a corresponding database 325A-C. As discussed above, eachserver 310 or 320 can correspond to a group of servers, and each ofthese servers can share a database or can have their own database.Though databases 315 and 325 are displayed logically as single units,databases 315 and 325 can each be a distributed computing environmentencompassing multiple computing devices, can be located within theircorresponding server, or can be located at the same or at geographicallydisparate physical locations.

Network 330 can be a local area network (LAN), a wide area network(WAN), a mesh network, a hybrid network, or other wired or wirelessnetworks. Network 330 may be the Internet or some other public orprivate network. Client computing devices 305 can be connected tonetwork 330 through a network interface, such as by wired or wirelesscommunication. While the connections between server 310 and servers 320are shown as separate connections, these connections can be any kind oflocal, wide area, wired, or wireless network, including network 330 or aseparate public or private network.

In some implementations, servers 310 and 320 can be used as part of asocial network. The social network can maintain a social graph andperform various actions based on the social graph. A social graph caninclude a set of nodes (representing social networking system objects,also known as social objects) interconnected by edges (representinginteractions, activity, or relatedness). A social networking systemobject can be a social networking system user, nonperson entity, contentitem, group, social networking system page, location, application,subject, concept representation or other social networking systemobject, e.g., a movie, a band, a book, etc. Content items can be anydigital data such as text, images, audio, video, links, webpages,minutia (e.g., indicia provided from a client device such as emotionindicators, status text snippets, location indictors, etc.), or othermulti-media. In various implementations, content items can be socialnetwork items or parts of social network items, such as posts, likes,mentions, news items, events, shares, comments, messages, othernotifications, etc. Subjects and concepts, in the context of a socialgraph, comprise nodes that represent any person, place, thing, or idea.

A social networking system can enable a user to enter and displayinformation related to the user’s interests, age / date of birth,location (e.g., longitude/latitude, country, region, city, etc.),education information, life stage, relationship status, name, a model ofdevices typically used, languages identified as ones the user is facilewith, occupation, contact information, or other demographic orbiographical information in the user’s profile. Any such information canbe represented, in various implementations, by a node or edge betweennodes in the social graph. A social networking system can enable a userto upload or create pictures, videos, documents, songs, or other contentitems, and can enable a user to create and schedule events. Contentitems can be represented, in various implementations, by a node or edgebetween nodes in the social graph.

A social networking system can enable a user to perform uploads orcreate content items, interact with content items or other users,express an interest or opinion, or perform other actions. A socialnetworking system can provide various means to interact with non-userobjects within the social networking system. Actions can be represented,in various implementations, by a node or edge between nodes in thesocial graph. For example, a user can form or join groups, or become afan of a page or entity within the social networking system. Inaddition, a user can create, download, view, upload, link to, tag, edit,or play a social networking system object. A user can interact withsocial networking system objects outside of the context of the socialnetworking system. For example, an article on a news web site might havea “like” button that users can click. In each of these instances, theinteraction between the user and the object can be represented by anedge in the social graph connecting the node of the user to the node ofthe object. As another example, a user can use location detectionfunctionality (such as a GPS receiver on a mobile device) to “check in”to a particular location, and an edge can connect the user’s node withthe location’s node in the social graph.

A social networking system can provide a variety of communicationchannels to users. For example, a social networking system can enable auser to email, instant message, or text/SMS message, one or more otherusers. It can enable a user to post a message to the user’s wall orprofile or another user’s wall or profile. It can enable a user to posta message to a group or a fan page. It can enable a user to comment onan image, wall post or other content item created or uploaded by theuser or another user. And it can allow users to interact (via theirpersonalized avatar) with objects or other avatars in an artificialreality environment, etc. In some embodiments, a user can post a statusmessage to the user’s profile indicating a current event, state of mind,thought, feeling, activity, or any other present-time relevantcommunication. A social networking system can enable users tocommunicate both within, and external to, the social networking system.For example, a first user can send a second user a message within thesocial networking system, an email through the social networking system,an email external to but originating from the social networking system,an instant message within the social networking system, an instantmessage external to but originating from the social networking system,provide voice or video messaging between users, or provide an artificialreality environment were users can communicate and interact via avatarsor other digital representations of themselves. Further, a first usercan comment on the profile page of a second user, or can comment onobjects associated with a second user, e.g., content items uploaded bythe second user.

Social networking systems enable users to associate themselves andestablish connections with other users of the social networking system.When two users (e.g., social graph nodes) explicitly establish a socialconnection in the social networking system, they become “friends” (or,“connections”) within the context of the social networking system. Forexample, a friend request from a “John Doe” to a “Jane Smith,” which isaccepted by “Jane Smith,” is a social connection. The social connectioncan be an edge in the social graph. Being friends or being within athreshold number of friend edges on the social graph can allow usersaccess to more information about each other than would otherwise beavailable to unconnected users. For example, being friends can allow auser to view another user’s profile, to see another user’s friends, orto view pictures of another user. Likewise, becoming friends within asocial networking system can allow a user greater access to communicatewith another user, e.g., by email (internal and external to the socialnetworking system), instant message, text message, phone, or any othercommunicative interface. Being friends can allow a user access to view,comment on, download, endorse or otherwise interact with another user’suploaded content items. Establishing connections, accessing userinformation, communicating, and interacting within the context of thesocial networking system can be represented by an edge between the nodesrepresenting two social networking system users.

In addition to explicitly establishing a connection in the socialnetworking system, users with common characteristics can be consideredconnected (such as a soft or implicit connection) for the purposes ofdetermining social context for use in determining the topic ofcommunications. In some embodiments, users who belong to a commonnetwork are considered connected. For example, users who attend a commonschool, work for a common company, or belong to a common socialnetworking system group can be considered connected. In someembodiments, users with common biographical characteristics areconsidered connected. For example, the geographic region users were bornin or live in, the age of users, the gender of users and therelationship status of users can be used to determine whether users areconnected. In some embodiments, users with common interests areconsidered connected. For example, users’ movie preferences, musicpreferences, political views, religious views, or any other interest canbe used to determine whether users are connected. In some embodiments,users who have taken a common action within the social networking systemare considered connected. For example, users who endorse or recommend acommon object, who comment on a common content item, or who RSVP to acommon event can be considered connected. A social networking system canutilize a social graph to determine users who are connected with or aresimilar to a particular user in order to determine or evaluate thesocial context between the users. The social networking system canutilize such social context and common attributes to facilitate contentdistribution systems and content caching systems to predictably selectcontent items for caching in cache appliances associated with specificsocial network accounts.

FIG. 4 is a block diagram illustrating components 400 which, in someimplementations, can be used in a system employing the disclosedtechnology. Components 400 can be included in one device of computingsystem 100 or can be distributed across multiple of the devices ofcomputing system 100. The components 400 include hardware 410, mediator420, and specialized components 430. As discussed above, a systemimplementing the disclosed technology can use various hardware includingprocessing units 412, working memory 414, input and output devices 416(e.g., cameras, displays, IMU units, network connections, etc.), andstorage memory 418. In various implementations, storage memory 418 canbe one or more of: local devices, interfaces to remote storage devices,or combinations thereof. For example, storage memory 418 can be one ormore hard drives or flash drives accessible through a system bus or canbe a cloud storage provider (such as in storage 315 or 325) or othernetwork storage accessible via one or more communications networks. Invarious implementations, components 400 can be implemented in a clientcomputing device such as client computing devices 305 or on a servercomputing device, such as server computing device 310 or 320.

Mediator 420 can include components which mediate resources betweenhardware 410 and specialized components 430. For example, mediator 420can include an operating system, services, drivers, a basic input outputsystem (BIOS), controller circuits, or other hardware or softwaresystems.

Specialized components 430 can include software or hardware configuredto perform operations for the creation and orchestration of virtualobjects, defined as a collection of components. Specialized components430 can include virtual object definition creator 434, reusable templatecreator 436, virtual object hierarchy manager 438, virtual objectinstantiator 440, and components and APIs which can be used forproviding user interfaces, transferring data, and controlling thespecialized components, such as interfaces 432. In some implementations,components 400 can be in a computing system that is distributed acrossmultiple computing devices or can be an interface to a server-basedapplication executing one or more of specialized components 430.Although depicted as separate components, specialized components 430 maybe logical or other nonphysical differentiations of functions and/or maybe submodules or code-blocks of one or more applications.

Virtual object definition creator 434 can create a definition of a newvirtual object by receiving a designation of which object in an objecthierarchy, maintained by virtual object hierarchy manager 438, the newvirtual object extends, defining additional or alternate components(container, data, controller, and/or template) or component portions forthe new virtual object; and connecting data and controller elements tofields of view states defined in the template. Additional details ondefining a virtual object are provided below in relation to FIG. 19 .

Reusable template creator 436 can define a template that can be used ina virtual object, by virtual object definition creator 434 or stored ina library of prefabricated templates usable by other systems to createvirtual objects. Each template can specify a state machine with one ormore view states (each defining which and how data elements of a virtualobject are displayed while that view state is active for the virtualobject) and contextual breakpoints for transitioning between which viewstate is the active view state. Additional details on defining areusable template are provided below in relation to FIG. 20 .

Virtual object hierarchy manager 438 can manage a hierarchy of virtualobject definitions, where each virtual object definition can inheritcomponents from its parent in the hierarchy. A base object definitioncan specify default features of virtual objects, allowing for virtualobjects to be quickly created by developers and have consistent andexpected behaviors for users. In some implementations, the hierarchy canspecify a set of native object definitions that extend the base object,for common types of virtual objects, such as people, places, events,social media posts, conversations, groups, photos, webpages, etc. Insome implementations, a special purpose object definition that extendsthe base object can define a virtual object for content items which maybe commonly used, imported from other platforms, and/or that are userdefined. In yet further implementations, a custom virtual objectdefinition that extends the base object can allow developers to createvirtual object definitions that inherit some basic content and/orfunctionality from the base object, allowing them to be used in anartificial reality environment, while providing full flexibility to thedeveloper to define and customize other aspects of the objects.Additional details on object hierarchies are provided below in relationto FIGS. 11 and 12 .

Virtual object instantiator 440 can invoke a constructor for a virtualobject defined by virtual object definition creator 434, with a set ofparameters required by the definition, to instantiate a version of thevirtual object in an artificial reality environment. For example, asystem can receive an identification of a context of an artificialreality device, such as a state of the artificial reality environmentaround the artificial reality device, a state of the artificial realitydevice, a state of the user of the artificial reality device, and/or astate of other factors determined in the world, and use a mapping ofsome of these contextual factors to virtual objects to determine whichvirtual object(s) to instantiate. The system can provide the virtualobject(s) to the artificial reality device, which can instantiate themby providing the appropriate contextual factors or other parameters(e.g., container, data, template, and/or controller parameters) theconstructor needs to instantiate an instance of the virtual object.

FIGS. 5A and 5B are conceptual diagrams 500 and 550 illustrating abreakdown of an object into components including a container, data, atemplate, and a controller. The container defines a volume, allocated tothe virtual object by an application controlling an artificial realityenvironment, into which the virtual object can write to provide output.In conceptual diagram 500, the container includes an outer container 502for the overall virtual object, a first inner container 504 for visualelements of the virtual object, and a second inner container 506 for UIelements of the virtual object. Examples of the data include thedisplayable (e.g., visual, auditory, haptic, etc.) elements that can beoutput by the virtual object; state or properties of the virtual object(e.g., configuration, settings, etc.); and meta data of the virtualobject (e.g., identifiers, location, name, semantic identifiers, etc.)Additional examples of virtual object data are provided below inrelation to FIGS. 13-18 . In conceptual diagram 500, the data includesexternal, dynamic data, such as data 508, that the virtual objectaccesses, internal data 510 that is local to the virtual object, andrelationships among data elements 512. The template 514 includesmultiple view states, where each view state defines how the virtualobject presents itself when that view state is active. A view state can,e.g., set a configuration of the container, map which data elements ofthe virtual object are displayed and where they are displayed within thecontainer, which UI elements are shown/hidden, or define other outputfor the virtual object. The template can act as a state machine whereeach view state is a state in the state machine and there is contextualbreakpoint logic that defines when the virtual object transitionsbetween the view states. A controller 516 can include logic specifyinghow the virtual object responds to user actions, determined world state/ context, input from system services and applications (e.g.,communication from network), etc. The portion 516 of the controllerillustrated in conceptual diagram 500 illustrates logic of thecontroller controlling how data element 508 plugs into an externalsource (e.g., if the virtual object 520 is a person object, controller516 can define how the person ID 508 is connected to an element in asocial graph - linking the person virtual object 520 to other people inthe social graph defined as “friends” of the person represented by thevirtual object 520).

The components of the virtual object can be combined, as shown by line518, to create the overall virtual object 520, which is shown in largerview in FIG. 5B. Thus, FIG. 5B illustrates the conceptual design of avirtual object, including a container 502 (with sub-containers 504 and506) defining a volume for the virtual object data elements, such asdata element 508, which are laid out according to an active view stateof a template 514, and where actions of the virtual object andconnections of the virtual object to external sources are controlled bycontroller, such as controller portion 516.

FIG. 6 is a conceptual diagram 600 illustrating an object in 3D spacedisplayed according to a view state from a template. Other conceptualdiagrams illustrated herein use 2D representations of virtual objectsand virtual object components for clarity. Conceptual diagram 600illustrates a 3D artificial reality environment to show that thesevirtual objects can be made up of 3D elements that exist in the 3Dartificial reality environment. In conceptual diagram 600, a virtualobject has an outer container 602 defining the overall volume into whichthe virtual object can write to display its elements. Conceptual diagram600 also shows bounding boxes 604 and 606 as two possible areas, withinthe volume 602, that each could be defined by a view state fordisplaying some of the virtual object’s data elements while in that viewstate. Conceptual diagram 600 further shows an origin point 608, whichthe view states can use as a point to define positions of data elementswithin the virtual object.

FIG. 7 is a conceptual diagram 700 illustrating a container component ofan object and an example implementation of the container component. Onthe left side of conceptual diagram 700, the container componentincludes an outer container element 702 defining the entire volume ofthe virtual object, a first inner container element 704 specifying aportion of the area defined by the outer container element 702 forgeneral content of the virtual object (e.g., visual 2D and 3D models),and a second inner container element 706 specifying a portion of thearea defined by the outer container element 702 for UI elements (e.g., aconsistent set of elements that enable a user to interact with virtualobjects in an expected manner, such as to perform common system actionslike Close and Share). On the right side of conceptual diagram 700, anexample implementation of the container component, as a person virtualobject, is shown where the outer container element 708 is shown (whichcan be presented to a user in some cases - such as when an object isselected to signify the object’s interactive boundary); the first innercontainer element 710 displays an avatar of the user represented by thevirtual object, and the second inner container element 712 shows astandard set of UI elements for interacting with a person virtualobject.

FIG. 8 is a conceptual diagram 800 illustrating a data component of anobject and an example implementation of the data component. On the leftside of conceptual diagram 800, data elements 802-806 are conceptuallyillustrated. External data elements 802 and 804 are dynamic dataelements imported to the data element from external sources such as asocial media platform. These data elements can live external to thevirtual object, can be periodically updated, or can have updates pushedto them as the external data elements change. For example, data element802 can be a user’s online status 816 as maintained in a first socialmedia platform and data element 804 can be a broadcast message 818,maintained in a second social media platform, that a user has selectedto display when her avatar appears in another’s artificial realityenvironment. These data elements can be updated as the user’s status orbroadcast message changes on the various social media platforms. Thevirtual object can also have internal data elements, such as dataelements 806, which may be set only a single time and/or may not link toany external source. For example, a person virtual object may have aname 810 which exists locally to the virtual object and is only set whenthe person virtual object is first created and may have an avatar 812,which is copied to the virtual object and exists locally, only beingupdated when the user updates the avatar through the virtual object. Thedata for a virtual object may have fields that link to an externalsource, but are not set in that source and thus remain empty. Forexample, the user corresponding to a person virtual object may not haveset a profile picture on a social media platform, and thus the profilepicture 814 data element may be empty. In some cases, the virtual objectcan define interactions or relationships among its data elements, shownas link 808. For example, a person virtual object could define set ofuser IDs that each specify the corresponding user profile on acorresponding social media platform.

FIGS. 9A and 9B are conceptual diagrams 900 and 950 illustrating atemplate component of a person object with multiple view states and anexample set of contextual breakpoint transitions between the viewstates. Conceptual diagram 900 illustrates five view states 902-910 of aperson virtual object. Each view state defines which data elements ofthe virtual object are displayed, and where and how they are displayedwithin the container of the virtual object, while that view state isactive. View state 902 is a glint view state where just an iconrepresentation of the person virtual object is displayed - such as in acontext where the virtual object has been inactive for a thresholdperiod of time. View state 904 is a non-surface view state, where a headportion of an avatar, of the person represented by the virtual object,is displayed - such as in a context where the virtual object is activebut is not attached to any surface. View state 906 is a horizontalsurface view state, where a complete avatar, of the person representedby the virtual object, is displayed - such as in a context where thevirtual object is active and is attached to a horizontal surface. Viewstate 908 is a vertical surface view state, where a flat representation(i.e., virtual photo) of a top portion of an avatar, of the personrepresented by the virtual object, is displayed - such as in a contextwhere the virtual object is active and is attached to a verticalsurface. View states 904-908 also include UI elements 912, which includea consistent set of UI controls for interacting with a person virtualobject in an active state. View state 910 is a holographic view state,where a live hologram, of the person represented by the virtual object,is displayed - such as in a context where there is an ongoingholographic call between the viewing user and the person represented bythe virtual object. View state 914 also includes UI elements 914, whichinclude a set of UI controls for controlling a holographic call.

Conceptual diagram 950 illustrates the view states 952-960, organizedinto a state machine with contextual breakpoints represented by thelines 962-992. Each of the lines 952-992 in conceptual diagram 950represents a contextual breakpoint, defined by an expression that, whenan active view state has a line leading away from it and the expressionfor that line evaluates to true, the view state at the end of the linebecomes the active view sate. As an example, the expressioncorresponding to any of transition lines 966, 972, 982, and 990 to viewstate 960, can be the context that a holographic call has begun with theperson represented by the person virtual object. Each of the expressionsfor lines 968, 984, 970, and 992 can be for the context that aholographic call ended and the previous view state before entering theview state 960 was the state at the end of that line. The expression forthe contextual breakpoint corresponding to line 962 can be the contextoccurring that a user selects the virtual object while the virtualobject is not anchored to a surface, the expression for the contextualbreakpoint corresponding to line 974 can be the context occurring that auser selects the virtual object while the virtual object is anchored toa vertical surface, and the expression for the contextual breakpointcorresponding to line 980 can be the context occurring that a userselects the virtual object while the virtual object is anchored to ahorizontal surface. The expression for the contextual breakpointcorresponding to lines 964, 976, and 978 can be the context occurringthat a user has not focused her attention on the virtual object for athreshold amount of time. The expression for the contextual breakpointcorresponding to line 986 can be the virtual object having been movedfrom a horizontal to a vertical surface and the expression for thecontextual breakpoint corresponding to line 988 can be the virtualobject having been moved from a vertical to a horizontal surface. Whileconceptual diagram 950 illustrates some example view states, with someexample contextual breakpoints, for a person virtual object, a personvirtual object could have more, less, or other view states and theseview states could have more, less, or other contextual breakpoints.

FIG. 10 is a conceptual diagram 1000 illustrating a controller componentof an object and an example implementation of the controller component.The controller component can identify a context for the virtual object,such as user inputs, interactions with external elements (e.g., otherobjects, services, network data, etc.), and a world-state (e.g., whereand how the virtual object is placed, a mode of the virtual object,other objects in the environment, people in the environment, etc.) Thecontroller can also define logic for how the virtual object responds toaspects of this context, which in some cases includes providing input tothe other components of the virtual object. Thus, on the upper left sideof conceptual diagram 1000, the elements of the controller 1002 areillustrated as elements 1002A-H which connect to the data, template, andcontainer components. Examples of the logic provided by the controllerinclude: object behaviors (e.g., how the object reacts to other objects,inputs, and events, triggers for the object to take correspondingdisplay, configuration, or other output actions, etc.), spatialbehaviors (e.g., how the object moves, reacts to surface placement, setsview states in different contexts, etc.), component management (e.g.,translating detected contexts and events into parameters or triggers forthe container, data, or template components), data management (e.g.,what data the virtual object retrieves or updates in response to giventriggers), notifications (e.g., how the virtual object surfaces detectedevents in the artificial reality environment), privacy and securityactions (e.g., what data and interfaces the virtual object makesavailable to the user, other virtual objects, and other entities and howthe virtual object protects its internal resources), and/or systemactions (e.g., changing states for power management, managing andrequesting resources, hardware access, interfacing with the OS or othercontrolling application, thread execution management, etc.) By definingthe controller as internal to the virtual object, the virtual objectbecomes self-contained through scoping, as the controller is executedwithin the scope of the virtual object, the controller can use onlyresources and parameters of that virtual object.

The controller can also include definitions for how the virtual objectplugs into other objects and systems. For example, the controllerelement 1002A defines a connection to a messaging system for a personvirtual object. When the system is provided with a notification of a newmessage in a message thread where the person represented by the personvirtual object is a participant, the controller can have logic to have amessage thread virtual object, which is imbedded as a data element inthe person virtual object, (illustrated here as virtual object 1004),displayed in the container. In some cases, this incoming message contextmay be a contextual breakpoint, which the controller passes to thetemplate, casing a messaging view state for showing the thread dataelement to become active.

FIG. 11 is a conceptual diagram 1100 illustrating an example objecthierarchy. Each object in the object hierarchy is an object definition.Some of these object definitions can be used by an application toinstantiate an instance of that object, by passing parameters (e.g.,passed to a constructor function) to specify specific data for thefields defined in the object definition. The objects hierarchy shown inconceptual diagram 1100 is an example, but the disclosed technologyincludes other hierarchies with additional, alternate, or fewer objectdefinitions.

The object hierarchy includes a root base object 1102, in a first level,from which all other objects descend. The base object includes variouslow-level and external connections, data element fields, and logic,which all other virtual objects can inherit. For example, the baseobject can include an ID field, UI data elements, functions fordisplaying a notification; basic physics and movement patterns forvirtual objects; privacy and security actions (e.g., what data andinterfaces the virtual object makes available to the user, other virtualobjects, and other entities and how the virtual object protects itsinternal resources); system actions (e.g., changing states for powermanagement, managing and requesting resources, hardware access,interfacing with the OS or other controlling application, threadexecution management, etc.); a standard template; etc. Additionalexamples of features that a base object can include are provided belowin relation to FIG. 13 .

In the second level extending the base object, the hierarchy includesset of native objects 1104, as special purpose object 1106, and a customobject 1108. These virtual objects in the second level initially inheritthe components of the base object, while adding, replacing, or removingsome functionality, external connections, data, logic, view states, etc.

The native objects 1104 can include a set of commonly used virtualobject definitions that various applications can instantiate. Becausethese objects A) inherit components from the base object and B) are usedby multiple applications across the artificial reality system, userswill have established usage patterns for these objects and theirfunctionality will be standard and expected. FIG. 12 is a conceptualdiagram 1200 illustrating an example native objects portion of theobject hierarchy. Examples of some of the native objects include aperson native virtual object 1202, a photo native virtual object 1204, aplace native virtual object 1206, a social media post native virtualobject 1208, an event native virtual object 1210, a group native virtualobject 1212, and a conversation native virtual object 1214. Additionalexamples of features of some native objects are provided below inrelation to FIGS. 14-18 . App objects 1110 each extend one of the nativeobjects 1104, allowing individual applications to create objectdefinition versions of the native object definitions, which add, modify,or remove functionality from the native object that object extends. Forexample, a person app object can extend the person native virtual object1202, allowing for that person app object to connect to an additional oralternate social media platform, provide a different type of avatardisplay, add additional communication connections for the person, defineadditional ways to notify or receive notifications from the representedperson, etc. This extension of app objects from the native objectsallows such extended objects to inherit the secure and consistentexperience users expect for particular types of native objects, whilegiving developers the freedom to customize the virtual objects to theirspecifications.

The special purpose object 1106 can include a custom container andtemplate for content items which may be commonly used, imported fromother platforms, and/or are user defined. Such a special purpose objectcan be a quasi-native catch-all object for content items for which usersshould have a consistent experience and for which developers should beable to easily create, but that do not fall into the other native objectcategories. Thus, a developer can extend the special purpose object, asone of special purpose objects 1112, to quickly create a useable objectwith standard functionality. For example, the special purpose objects1112 shown in conceptual diagram 1100 include a clock 3D model, asticker decal, a weather widget, and a confetti animation. The confettianimation, for example, may simply be created by defining the virtualobject as an extension of the special purpose object 1106 and specifyinga 3D animation data element, to use with the default special purposeobject 1106's container, other data, templates, and controller. Asanother example, some of the special purpose objects 1112 can beautomatically crated by importing existing content from another system.For example, a variety of 3D content may exist for another platform,such as user generated AR content, 3D models of shopping items from aretailer, features scanned from the world by a mapping system, etc.These items can be paired with a set of parameters used by that systemand then created as items in special purpose objects 1112, inheritingthe necessary components to instantiate and use them in an artificialreality environment.

The custom object 1108 and its subsequent extension objects 1114represent additional objects that developers can create that inheritsome basic content and/or functionality from the base object 1102,allowing them to be used in an artificial reality environment, whileproviding full flexibility to the developer to define and customizeother aspects of the objects. A developer can create one or moveversions of the branch of custom object 1108 and its extensions 1114,e.g., defining any custom container, data, template, and/or controllercomponent or component’s elements. As discussed above, a virtual objectcan be a compound object, thus a developer can still easily create acomplex and flexible custom object branch, that still has elements thatare familiar to users, by including any of the native objects 1104, appobjects 1110, or special purpose objects 1112, as data elements of acustom object.

FIG. 13 is a conceptual diagram 1300 illustrating an example descriptionof a base object (e.g., base object 1102). Conceptual diagram 1300includes components of a base virtual object including a data component1302, a template component 1304, a container component 1306, and acontroller component 1308. Each of these components includes some basicelements that can be passed to other virtual objects that extend thisbase virtual object. The data component 1302 includes a default set ofmeta data and content assets. The template component 1304 includes adefault set of view states and contextual breakpoint definitions. Thecontainer component 1306 includes a defaults set of container manifestparameters for positioning and orientating a container. The controllercomponent 1308 includes a default set of data management logic, adefault set of user action logic, and a default set of system actionlogic.

FIG. 14 is a conceptual diagram 1400 illustrating an example descriptionof a native person object (e.g., native person object 1202). The nativeperson object extends to base object described in conceptual diagram1300, inheriting its data, template, container, and controllercomponents, and includes additions such as: data 1402 with additionalmeta data and additional content assets and controller 1408 withadditional user actions and additional system actions.

FIG. 15 is a conceptual diagram 1500 illustrating an example descriptionof a native place object (e.g., native place object 1206). The nativeplace object extends to base object described in conceptual diagram1300, inheriting its data, template, container, and controllercomponents, and includes additions such as: data 1502 with additionalmeta data and additional content assets, template 1504 with additionalview sates and contextual breakpoints, and controller 1508 withadditional data management, user actions, and additional system actions.

FIG. 16 is a conceptual diagram 1600 illustrating an example descriptionof a native photo object (e.g., native photo object 1204). The nativephoto object extends to base object described in conceptual diagram1300, inheriting its data, template, container, and controllercomponents, and includes additions such as: data 1602 with additionalmeta data and additional content assets, template 1604 with additionalview sates and contextual breakpoints, and controller 1608 withadditional data management, user actions, and additional system actions.

FIG. 17 is a conceptual diagram 1700 illustrating an example descriptionof a native conversation object (e.g., native conversation object 1214).The native conversation object extends to base object described inconceptual diagram 1300, inheriting its data, template, container, andcontroller components, and includes additions such as: data 1702 withadditional meta data and additional content assets, template 1704 withadditional view sates and contextual breakpoints, and controller 1708with additional data management and user actions.

FIG. 18 is a conceptual diagram 1800 illustrating an example descriptionof a native post object (e.g., native post object 1208). The native postobject extends to base object described in conceptual diagram 1300,inheriting its data, template, container, and controller components, andincludes additions such as: data 1802 with additional meta data andadditional content assets, template 1804 with additional view sates andcontextual breakpoints, and controller 1808 with additional datamanagement, user actions, and additional system actions.

Those skilled in the art will appreciate that the components illustratedin FIGS. 1-18 described above, and in each of the flow diagramsdiscussed below, may be altered in a variety of ways. For example, theorder of the logic may be rearranged, substeps may be performed inparallel, illustrated logic may be omitted, other logic may be included,etc. In some implementations, one or more of the components describedabove can execute one or more of the processes described below.

FIG. 19 is a flow diagram illustrating a process 1900 used in someimplementations for defining a virtual object with a modular designprocess. In various implementations, process 1900 can be performed on aclient device of a developer specifying the components of a virtualobject or on a server providing service to such a client device.

At block 1902, process 1900 can receive a definition of an inheritance,in an object hierarchy, for a new object. The hierarchy can be acollection of virtual objects where each virtual object, other than aroot virtual object of the hierarchy, is a child of one other virtualobject in the hierarchy, as discussed above in relation to FIGS. 11 and12 . Each virtual object can initially inherit the components of itsparent virtual object, which can then be replaced or modified withprocess 1900. Block 1902 can include specifying which object in thehierarchy is the parent of the new virtual object, thus defining theinitial set of components inherited from that parent virtual object. Forexample, when virtual object B is created to inherit from (also referredto as “extending”) virtual object A, and virtual object A has a templateT, virtual object B automatically has template T, which can be modifiedor overridden at block 1912 discussed below.

At block 1904, process 1900 can specify additional and/or alternatecontainer, data, template and/or controller components from thoseinherited from the new virtual object’s parent virtual object in thehierarchy. In various implementations, process 1900 can provide detailsof the inherited object components which a user can drill into to changeindividual values or replace with completely different components. Insome implementations, selecting a different component can includeproviding a definition for a new component or selecting an existingcomponent, e.g., from a component prefab library.

A virtual object container (also referred to as an “augment”) is createdwhen the virtual object is instantiated by an application sending arequest to the controlling application with the container manifest,specifying basic properties of the container such as types of surfacesor locations where the container can be placed, how the container willdisplay itself in different modes, initial location, size, andorientation, etc. In some implementations, a container can have aspecific pre-defined type (e.g., person, 2D media, post, event, or 3Dmodel), and a default manifest for that type can be used. Thecontrolling application creates a container based on the manifest andprovides a handle to it back to the requesting application. Thisrequester can then write content into the container (including writingdifferent versions for different display properties specified in themanifest). The controlling application can maintain control of where thecontainer can be displayed and which display properties are invoked,according to allowed positions and display properties specified in themanifest. Additional details on containers are provided in U.S. Pat.Application No. 17/008,478, titled “Artificial Reality Augments andSurfaces,” and filed on Aug. 31, 2020, which is herein incorporated byreference in its entirety. Specifying changes to or an alternatecontainer, at block 1906, can include changing or defining a set ofparameters (i.e., a “manifest”) that an application instantiating thevirtual object will use to request, from an application controlling anartificial reality environment (e.g., operating system, shellapplication, etc.), a volume (2D or 3D) to write into. Thus, at block1906, process 1900 can receive a manifest definition or updates to themanifest inherited from the parent virtual object, e.g., by changing thetype, size, possible anchor types or locations, etc.

The new virtual object’s data can include, e.g., visual elements, statedefinitions, parameters and meta-data, attachable Uls, other objectsused inside this object (i.e., “compound objects”), etc. While a virtualobject can have any number and type of data elements, an example virtualobject can have data defining various 2D and 3D models it can display inits container; parameters such as an ID, name, corresponding socialmedia entity ID, status, originating application, a set of UI elements,etc. See additional example virtual objects with some of their datadefinitions in FIGS. 8 and 13-18 . Specifying additional and/oralternate data, at block 1908, for the new virtual object can includeadding new data to that inherited from the parent virtual object,removing data from that inherited from the parent virtual object, and/orredefining data elements inherited from the parent virtual object. Invarious implementations, the specified data can be a definition of thelocation of the data, e.g., a link, file storage location, or otherstorage path. An application that will instantiate the virtual objectcan supply the given data at the defined data location (e.g., uponinstallation or when the virtual object is retrieved) or theinstantiation process can include retrieving the data from a remotelocation specified by the storage path. In some implementations, thedata can be dynamic, such that as it changes in the remote location,those changes are reflected in the virtual object.

A virtual object’s controller can define logic and rules specifying howthe virtual object interacts with external elements (e.g., otherobjects, services, network data, etc.) and responds to a determinedworld-state / context and user input. In various implementations,examples of logic the controller can define include: object behaviors(e.g., how the object reacts to other objects, inputs, and events;triggers for the object to take corresponding display, configuration, orother output actions), spatial behaviors (e.g., how the object moves,reacts to surface placement, sets view states in different contexts,etc.), component management (e.g., translating detected contexts andevents into parameters or triggers for the container, data, or templatecomponents), data management (e.g., what data the virtual objectretrieves or updates in response to given triggers), notifications(e.g., how the virtual object surfaces detected events in the artificialreality environment), privacy and security actions (e.g., what data andinterfaces the virtual object makes available to the user, other virtualobjects, and other entities and how the virtual object protects itsinternal resources), and/or system actions (e.g., changing states forpower management, managing and requesting resources, hardware access,interfacing with the OS or other controlling application, threadexecution management, etc.) The controller can also include definitionsfor how the virtual object plugs into other objects and systems. Forexample, a controller for a person virtual object can define aconnection (e.g., via a person identifier) to an entity of a socialmedia platform’s social graph. Through this connection, the personvirtual object can be associated with friends, interests, events, etc.as defined in the social graph and the virtual object can be updatedaccording to events occurring on the social platform. By defining thecontroller as internal to the virtual object, the virtual object becomesself-contained through scoping, as the controller is executed within thescope of the virtual object, the controller can use only resources andparameters of that virtual object. Specifying the virtual object’scontroller at block 1910 can include changing any given ones of therules / logic inherited from the parent virtual object, specifyingadditional rules / logic, and/or removing inherited rules / logic.

A template component of a virtual object can include multiple viewstates, where each view state defines how the virtual object presentsitself - such as by setting a configuration of the container, mappingwhich data elements of the virtual object are displayed and where withinthe container, which UI elements are active, or other output for thevirtual object. The template can act as a state machine where each viewstate is a state in the state machine and there is contextual breakpointlogic that defines when the virtual object transitions between the viewstates. Additional details on view states and contextual breakpoints arediscussed above in relation to FIGS. 9A and 9B and below in relation toFIG. 20 . At block 1912, process 1900 can specify an alternate templateor modifications for an inherited template. This can include replacing,removing, or adding view states and/or changing contextual breakpointsfor transitioning between the view states in a template inherited fromthe parent virtual object or receiving a selection of a prefab template(e.g., defined using process 2000 discussed below). In some cases,process 2000, described below, can be used to define a template for anew virtual object.

At block 1914, process 1900 can define connections between thecomponents specified at block 1904. For example, this can includespecifying which objects from block 1908 map into view state fields fromblock 1912; which logic or rules from the controller defined at block1910 map to elements of the container, data, or template components;which view states from block 1912 map to which container configurationsdefined at block 1906, etc. Block 1914 can also include fillingmeta-data fields in the template (e.g., placement, name, ID, etc.) orhooking the virtual object to live data sources defined by theconnections in the controller. Process 1900 can then store the definedvirtual object for future instantiation before it ends.

FIG. 20 is a flow diagram illustrating a process 2000 used in someimplementations for constructing a reusable template. In variousimplementations, process 2000 can be performed on a client device of adeveloper specifying a virtual object template or on a server providingservice to such a client device. In some cases, process 2000 can beperformed as a sub-process of process 1900, e.g., at block 1912. Thetemplate system can adjust performance limitations on a per-templatebasis, while reducing the payload of containers to make them instant andperformant.

At block 2002, process 2000 can define one or more view states for atemplate, where each view state defines how a virtual object presentsitself - such as by setting a configuration of the container, mappingwhich data elements of the virtual object are displayed and where withinthe container, which UI elements are active, or other output for thevirtual object. Example view states that can be in a template are: a“glint” view state showing only an icon representation of the virtualobject, a “preview” view state showing low resolution or simplifiedversions of content from a selected view state, a “notify” view stateproviding information on an event related to the virtual object (e.g., amessage from a network source arrived in relation to the virtualobject), a “selected” view state showing a full 3D version of thevirtual object, a “quick actions” view state showing UI to take commonactions on the virtual object, a “horizontal surface” view statedisplaying the virtual object as a 2D panel, a “vertical surface” viewstate showing the virtual object as a 3D object resting on a surface,etc. Each view state can specify fields specifying a data element of thevirtual object to output and parameters for that output such as a size,orientation, location within the virtual object’s container, volume,mode, etc. In some cases, one or more of the view states can be selectedfrom an existing library of view states or from the templates ofexisting virtual objects.

At block 2004, process 2000 can define directed links with correspondingcontextual breakpoint expressions between the view states defined inblock 2002. A template can act as a state machine where each view stateis a state in the state machine and there is contextual breakpoint logicthat defines when the virtual object transitions between the viewstates. A contextual breakpoint can be an expression that is evaluatedbased on determined context of the virtual object, such as what type ofsurface or anchor point it is attached to, a mode of the artificialreality device displaying the virtual object, user inputs, identifiedsystem events, etc. Each contextual breakpoint can define a directedlink between two view states in the template, and when A) a view stateat the beginning of such a link is active and B) the contextualbreakpoint expression evaluates to true, the view state at the end ofthe link can become the active view state. Each contextual breakpointcan be defined with a directed link between two view states and canspecify binary values for whether certain contextual features, events,system status, etc. occur, along with combination logic (e.g., AND, OR,NOT, XOR, etc.) between these values, to define expression that canoverall evaluate to true or false depending on circumstances identifiedby an artificial reality device.

While any block can be removed or rearranged in various implementations,block 2006 is shown in dashed lines to indicate there are specificinstances where block 2006 is skipped. At block 2006, process 2000 cansave the defined template as a reusable prefabrication or “prefab.” Thiscan include storing the template locally, sharing the template withanother user, or saving the template to a library (e.g., cloud based)for future incorporation in a virtual object. Following block 2006 (orblock 2004 if block 2006 is skipped) process 2000 can end.

Reference in this specification to “implementations” (e.g., “someimplementations,” “various implementations,” “one implementation,” “animplementation,” etc.) means that a particular feature, structure, orcharacteristic described in connection with the implementation isincluded in at least one implementation of the disclosure. Theappearances of these phrases in various places in the specification arenot necessarily all referring to the same implementation, nor areseparate or alternative implementations mutually exclusive of otherimplementations. Moreover, various features are described which may beexhibited by some implementations and not by others. Similarly, variousrequirements are described which may be requirements for someimplementations but not for other implementations.

As used herein, being above a threshold means that a value for an itemunder comparison is above a specified other value, that an item undercomparison is among a certain specified number of items with the largestvalue, or that an item under comparison has a value within a specifiedtop percentage value. As used herein, being below a threshold means thata value for an item under comparison is below a specified other value,that an item under comparison is among a certain specified number ofitems with the smallest value, or that an item under comparison has avalue within a specified bottom percentage value. As used herein, beingwithin a threshold means that a value for an item under comparison isbetween two specified other values, that an item under comparison isamong a middle-specified number of items, or that an item undercomparison has a value within a middle-specified percentage range.Relative terms, such as high or unimportant, when not otherwise defined,can be understood as assigning a value and determining how that valuecompares to an established threshold. For example, the phrase “selectinga fast connection” can be understood to mean selecting a connection thathas a value assigned corresponding to its connection speed that is abovea threshold.

As used herein, the word “or” refers to any possible permutation of aset of items. For example, the phrase “A, B, or C” refers to at leastone of A, B, C, or any combination thereof, such as any of: A; B; C; Aand B; A and C; B and C; A, B, and C; or multiple of any item such as Aand A; B, B, and C; A, A, B, C, and C; etc.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Specific embodiments and implementations have been described herein forpurposes of illustration, but various modifications can be made withoutdeviating from the scope of the embodiments and implementations. Thespecific features and acts described above are disclosed as exampleforms of implementing the claims that follow. Accordingly, theembodiments and implementations are not limited except as by theappended claims.

Any patents, patent applications, and other references noted above areincorporated herein by reference. Aspects can be modified, if necessary,to employ the systems, functions, and concepts of the various referencesdescribed above to provide yet further implementations. If statements orsubject matter in a document incorporated by reference conflicts withstatements or subject matter of this application, then this applicationshall control.

1. A method of defining a virtual object, for access by an applicationprogram executed on a processing system, the method comprising:creating, using one or more memories and one or more processors, a datastructure stored in a computer system memory, the data structureincluding information used by the application program executed on theprocessing system and including: a data component defining multipleattribute data elements, the attribute data elements including at leastvisual elements, parameters, and links to external data; a manifest fora container component that defines a volume that the virtual object isauthorized to write into; a controller component defining logiccontrolling how the virtual object responds to contexts, the contextsincluding one or more of: user actions, a determined world state,outputs from system services, outputs from other applications, or acombination thereof; and a template component defining multiple viewstates with contextual breakpoints specifying transitions between themultiple view states.
 2. The method of claim 1, wherein one or more ofthe data component, the manifest for the container component, thecontroller component, and/or the template component is based on acorresponding component from a parent object that the virtual objectextends in an object hierarchy; and wherein at least one of the datacomponent, the manifest for the container component, the controllercomponent, and/or the template component is modified from the version inthe parent object in the object hierarchy.
 3. The method of claim 1,wherein one or more of the data component, the manifest for thecontainer component, the controller component, and/or the templatecomponent is based on a corresponding component from a parent objectthat the virtual object extends in an object hierarchy.
 4. The method ofclaim 3: wherein the object hierarchy includes at least a root objectfrom which all other objects in the object hierarchy descend; andwherein the object hierarchy includes at least a set of native objectsdefining at least a native person object and a native photo object. 5.The method of claim 3, wherein the object hierarchy includes at least aroot object from which all other objects in the object hierarchydescend; and wherein the object hierarchy includes at least a specialpurpose object defining a custom container component and a customtemplate component for content items imported from another platform,wherein the special purpose object inherits functionality from the rootobject permitting instances of the special purpose object, each with animported content item, to exist in an artificial reality environment. 6.The method of claim 1, wherein the controller component further defineslogic that causes the virtual object to react to one or more of: whichother people or objects are in the vicinity, how the virtual object isgrouped with or linked to other objects, a state associated with acurrent user, or a combination thereof.
 7. The method of claim 1,wherein the controller component includes definitions of how the virtualobject is associated with other virtual objects.
 8. The method of claim1, wherein the controller component includes connection specificsdefining connections between the virtual object and external servicesand defining how the virtual object responds to events from thoseexternal services.
 9. The method of claim 1, wherein each of themultiple view states, when that view state is the active view state forthe virtual object, controls what data elements the virtual object ispresenting and how the virtual object is presenting those data elements.10. The method of claim 1, wherein the template component is a statemachine with each view state being a state of the state machine and eachcontextual breakpoints being an expression that, for a particular stateand when the expression evaluates to true, causes a transition from theparticular state to another state.
 11. A non-transitory memory forstoring a definition of a virtual object, for access by an applicationprogram executed on a processing system, the memory comprising: a datastructure stored in the memory, the data structure including informationused by the application program and including: a data component definingmultiple attribute data elements, the attribute data elements includingat least visual elements, parameters, and links to external data; amanifest for a container component that defines a volume that thevirtual object is authorized to write into; a controller componentdefining logic controlling how the virtual object responds to contexts,the contexts including one or more of: user actions, a determined worldstate, outputs from system services, outputs from other applications, ora combination thereof; and a template component defining multiple viewstates with contextual breakpoints specifying transitions between themultiple view states.
 12. The non-transitory memory of claim 11, whereinone or more of the data component, the manifest for the containercomponent, the controller component, and/or the template component isbased on a corresponding component from a parent object that the virtualobject extends in an object hierarchy; and wherein at least one of thedata component, the manifest for the container component, the controllercomponent, and/or the template component is modified from the version inthe parent object in the object hierarchy.
 13. The non-transitory memoryof claim 11, wherein one or more of the data component, the manifest forthe container component, the controller component, and/or the templatecomponent is based on a corresponding component from a parent objectthat the virtual object extends in an object hierarchy.
 14. Thenon-transitory memory of claim 13: wherein the object hierarchy includesat least a root object from which all other objects in the objecthierarchy descend; and wherein the object hierarchy includes at least aset of native objects defining at least a native person object and anative photo object.
 15. The non-transitory memory of claim 13, whereinthe object hierarchy includes at least a root object; and wherein theobject hierarchy includes at least a special purpose object defining acustom container component and a custom template component for contentitems imported from another platform, wherein the special purpose objectinherits functionality from the root object permitting instances of thespecial purpose object, each with an imported content item, to exist inan artificial reality environment.
 16. The non-transitory memory ofclaim 11, wherein the controller component further defines logic thatcauses the virtual object to react to one or more of: which other peopleor objects are in the vicinity, how the virtual object is grouped withor linked to other objects, a state associated with a current user, or acombination thereof.
 17. The non-transitory memory of claim 11, whereinthe controller component includes definitions of how the virtual objectis associated with other virtual objects.
 18. The non-transitory memoryof claim 11, wherein the controller component includes connectionspecifics defining connections between the virtual object and one ormore external services and defining how the virtual object responds toevents from those one or more external services.
 19. A computing systemstoring a definition of a virtual object, for access by an applicationprogram executed on the computing system, the computing systemcomprising: one or more processors; and one or more memories storing thedefinition of the virtual object including: a data component definingmultiple attribute data elements, the attribute data elements includingat least visual elements, parameters, and links to external data; amanifest for a container component that defines a volume that thevirtual object is authorized to write into; a controller componentdefining logic controlling how the virtual object responds to contexts,the contexts including one or more of: user actions, a determined worldstate, outputs from system services, outputs from other applications, ora combination thereof; and a template component defining multiple viewstates with contextual breakpoints specifying transitions between themultiple view states.
 20. The computing system of claim 19, wherein eachof the multiple view states, when that view state is the active viewstate for the virtual object, controls what data elements the virtualobject is presenting and how the virtual object is presenting those dataelements.